DOCOHENT EESOHE 

2D 119 135 CS 002 442 



AUTHOR 
TITLE 



INSTITOTION 



PUB DATE 
NOTE 



Gavigan, Patrick; Heseaann, John 

Indiana Educational Needs Assessment Project: An 

Introduction to the Analysis of Correlates of Reading 

Skills in Indiana Public Schools. 

Indiana State Dept. of Public Instruction, 

Indianapolis. Div. of Innovative and Exemplary 

Education. 

Nov 75 

37p.; Paper presented at the Annual Heeting of the 
American Educational Research Association (San 
Francisco, California, April 1 9-23, 1 976) 



EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



MF-$0.83 HC-$2.06 Plus Postage 

♦Educational Assessment; *Educational Needs; 

Elementary Secondary Education; Evaluation Needs; 

♦Program Evaluation; Reading; *Reading Achievement; 

Reading Programs; Reading Research; *Reading 

Skills 

Indiana; *Indiana Educational Needs Assessment 
Project 



ABSTRACT 

This report is part of an overview of the 
aspirations, methods, and summary results of associational analyses 
of Indiana assessment data. The study was based on the recognition of 
policy makers' needs to understand the causes of existing conditions 
in their schools and to be able to evaluate compound implications of 
their actions. A survey data base was used and an effort was made to 
select variables which corresponded directly to present policy 
questions and which could be supported by a review of recent 
literature. Scaling was kept to a minimum and no a priori assumptions 
were formulated about what analyses might signify. The first section 
examines the background, purposes, methods, procedures, and major 
results obtained by a reanalysis of the 1973 assessment data. The 
major objectives of this study were to determine which aspects of the 
instructional process are most effective in increasing reading 
performance, and which of those important characteristics which 
govern reading ability could be altered within the context of the 
educational community. The second section is a summary of preliminary 
investigations. The third section is a summary of results with the 
student as the unit of analysis. The fourth section is a summary of 
results with the school as the unit of analysis. The two attachments 
include variable definitions and selected references. (TS) 
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PREFACE 



This report is an overview of the aspirations, methods, and 
summary results of associational analyses of Indiana assessment 
data. It is the first of two volumes to be produced, and will 
contain only summaries of major questions brought forth by this 
investigation. The complete narrative is more technical in nature, 
and rather than run the risk of not communicating with our primary 
audiences at all, we have chosen to publicize introductory thoughts 
under a separate cover. 

Our concern for the practical consumers of this investigation 
illustrates a dilemma confronting educational researchers. How 
does one demonstrate genuine pedagogical concern, fundamental to the 
effectiveness of any policy research, and still meet the specifica- 
tions of inferential statistical models? For example, when 
variables are scaled and/or combined to define indices, or to fit 
distribution assumptions, they may lose the direct interpretation 
necessary for policy decision-making. Also, when relationships are 
uncovered in survey data, the researcher is open to the charge of 
"data-snooping" for relationships which may be peculiar to the sampl 
surveyed . 

This study was conducted because we recognize policy makers' 
needs: 1) to understand multiple "cause" of existing conditions 
in their schools, and 2) to be able to evaluate compound implica- 
cations of their actions. On the other hand, we want to stress 
the critical need for more discussion within the research community 
of how higher-level analysis models might be more effectively 
applied toward policy-making, given the limitations of a data base 
derived primarily through non-experimental methods. 

Since any discussion of the paucity of theory is easily mis- 
interpreted by researchers, the point must be made at the outset 
that our position is that it is desirable and possible to formulate 
educational policies which are empirically informed and logically 
warranted. In pursuit of this notion we chose to work from a survey 
data base only, since it is doubtful that any other means of system- 
atic data collection on a large scale will be feasible in the 
immediate future. An effort v/as made to select variables which 
correspond directly to present policy questions, and also which 
could be supported by a review of recent literature. Scaling was 
kept to a minimum* This was done so that variables would maintain 
'a degree of face value and would correspond to those used to report 
this data basfe in descriptive fashion. Finally, we did not formu- 
late any a priori assumptions about what these analyses might 
signify other than a belief they might provide greater insight into 
our earlier findings. We tested no hypotheses, explicit or other- 
wise . 

For those readers whose interests lie in the field of reading, 
our concern about an inadequate theory to explain these analyses ' 
resulted in the reading process being treated more as a vehicle for 
analysis than as the subject under investigation. Although we offer 



findings which appear salient to the reading specialist, we treat 
them as impressions only. No attempt at thorough documentation of 
related pedagogical issues has been made for present purposes. In- 
stead, we sought to highlight findings at this time as they compare 
with other well-known investigations of a similar methodoloqical 
persuasion. 

By tackling the broader issue of overall school effectiveness, 
we accept some inherent criticisms. For instance, possible argu- 
ments stemming from issues such as the criteria for selection and 
scaling of variables, or the value of "R2" for drawing policy impli- 
cations m the first place, are duly noted. On the other hand, 
these possible concerns did not outweigh our need to take a pre- 
determined path for comparative purposes. Investigations which 
employed similar statistical techniques have forced broader social 
issues to impact heavily upon school policy, while key issues re- 
volving around quality instruction continue to be resolved upon 

show and tell" documentation or left entirely to intuitive iudq- 
ments. •' 



While there are hurdles to overcome, there is value in pursuing 
causal inference models based on assessment/evaluation-type data. 
For example, no one can argue that studies which take a macroview 
of educational interventions producing descriptive statements about 

current status of affairs" best serve the information needs of 
the community at large. But is this the case with teachers? It 
would appear that school official-teacher dialogue could benefit 
by studies which demonstrate administrators' concern for underlying 
factors associated with improved instructional programs. In any 
case, such an exercise would be of benefit to the researcher, in 
his recognition of the need to strike a more reasoned balance 
between the indepth richness provided by small " samples and the 
ability for generalization provided by large ones. 

For those researchers who might wish to employ association 
analysis techniques as a portion of future analysis plans, we 
recommend the analysis be focused on these questions: 

1) To what degree do specific school programs, instruc- 
tional practices, etc., influence school outcomes? 

2) In what ways? 

3) Who might benefit most by altering school programs or 
practices? 

The analytical model we found most useful in pursuing these 
questions partitions the amount of variation among student scores 
accounted for by each of the following components in step-wise 
fashion : 

* 

a) Individual student background factors; 

b) Background factors, and other student characteristics 
such as attitude; 
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c) Background factors: other characteristics, 
and conununity-related school factors such 
as location, income levels, physical plant, 
etc . ; 

d) Background factors; other characteristics; 
community variables, and instructional program 
characteristics , 

We suggest that the student is the logical unit of ana'lysis. A 
partition of variance separated for white, black and other races 
is recommended as well. Although not a possibility for this study, 
we would suggest that instructional treatment variables be broken 
out further to include characteristics of individual teachers and 
other school input characteristics by individual student. Data 
which would enable the investigator to match individual achievement 
with individual resources might provide insight into questions 
such as "Do te chers make a difference? Is nonschool learning 
intervention universal in its effects on the formal education 
process?" 

Thanks are extended to Ron Lac is , a 2nd grade student at 
Mary Evelyn Castle Elementary School in Indianapolis, for his 
award-winning cover design. 

Dr. Patrick Gavigan 

Needs Assessment Coordinator 

John Hesemann 
Statistician 
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HIGHLIGHTS OF ASSESSMENT DATA REANALYSIS 

This material highlights the background, purposes, methods, 
procedures, and major results obtained by the reanalysis of the 
1973 assessment data. A question and answer format is used to 
introduce major points. 

V7hat is the purpose of analyzing the correlates of reading 
achievement ? 

The intent of this study^is to uncover those factors which 
appear to explain why some students read better than others. 
The major objectives of this study are to determine which aspects 
of the instructional process are most effective in increasing read- 
ing performance and which of those important characteristics which 
govern reading ability could be altered within the context of the 
educational community. By analyzing correlates of achievement, 
this study also serves future needs assessment efforts. Predictors 
of the genre developed by this investigation will facilitate a 
more reasoned process for identifying school reading programs most 
in need of modification. 

How does this study differ from past reports ? 

The intent of the initial analysis of assessment data gathered 
in 197 3 was to accurately describe the status of student attainment 
of a cross section of goals for public education in Indiana. Written 
for lay audiences, earlier reports sought only to picture existing 
levels of student performance. The present study is designed to 
explore the kinds of statistical relationships which may be inferred 
between a student's performance of selected reading tasks and (a) 
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the type of school he attends, (b) his socioethnic background, 
(c) his attitudes toward school and the future, and (d) indi- 
cators of the type of instructional programs employed in the 
student's school. 

To treat these questions at other than a descriptive level 
required the application of associational models of data analysis 
not always familiar to needs assessment audiences. Unfortunately, 
the application of association analysis models to survey data is 
an issue in itself. As a result, methodology must be carefully dis- 
cussed before reporting the results of these analyses. 

This study differs from past treatments of this data in the 
manner in v/hich the data was collected prior to analysis. Earlier 
reports dealt with the percentage of correct responses for each 
item. These percentages for all items contributing to an objective 
were averaged to reflect a score for each of four reading objectives. 
The same procedure was followed to report scores for major subpopu- 
lations — divided by region, community type and size, sex, race, 
and levels of parental education. This was done at each of three 
grade levels. On the other hand, exploration of the ways in which 
results differ from one student to another requires that data be 
collected across items to reflect scores for each individual student 
rather than an average score across items for an entire group. The 
criterion of performance (dependent variable) is then altered to 
reflect a single score for each student over a cluster of items rather 
than the percentage of responses on an item-by-item basis. 



Why was reading singled out for this type of analysis ? 

Because of the magnitude of data gathered, as well as the 
technical tasks required to create an appropriate criterion (depen- 
dent variable) for analysis of correlates, effort had to be 
limited to one area of interest. Reading was selected for several 
reasons. Since reading is a high priority issue both within the 
state and the nation, the results of the study should be of interest 
to a broad spectrum of clients. It can also be argued that results 
pertaining to correlates, of reading can be generalized to other 
areas of academic performance more readily than would be the case 
if the criterion were based upon another academic skill. 

From a methodological point of view, reading offered the widest 
range of items from which to construct a criterion for analysis. 
The fact that many of the reading items have been utilized by the 
National Assessment of Educational Progress (NAEP) also prompted the 
focus on reading. For instance, the use of NAEP items diminished 
the need to establish the external validity of the data. In addi- 
tion, the analysis of Indiana data based on NAEP items could enhance 
the transportability of these findings to a national audience-; One 
sample of how these analyses could prove useful to national inter- 
ests can be demonstrated by the following issue: 

National Assessment has been promoting the notion that the 
correlates of achievement are substantially different when using 
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performance rather than general ability as the criterion for 
measurement. ^ 

Where did the data come from ? 

Data presented in this report was gathered in the spring 
of 1973 at the request of the Division of Innovative and Exem- 
plary Education (ESEA Title III) of the Indiana Department of 
Public Instruction. The study was conducted pursuant to Public 
Law 89-10, which calls on each state to establish priorities for 
innovative programs on the basis of assessed learner performance 
as a function of systematically established statewide goals. 

The data represents the responses of more than 17,500 students. 
These students were included in a 6 per cent stratified, cluster 
sample of students who were enrolled at the 4th and 12th grade 
levels in the Indiana public schools. The sample was drawn in three 
stages. First, school corporations were partitioned by (1) four 
categories of region, (2) five categories of community type, and 
(3) two categories of size, yielding 40 cells — new classification 
categories. Next, corporations were selected at random from each 
cell until the total enrollment of corporations selected from a 
cell was approximately 6 per cent of the total enrollment of all cor- 
porations in the cell. In corporations with a small number of 
schools/ all schools were represented in the sample. In corporations 
with a large number of schools, a subset of schools was chosen in 
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National Assessment of Educational Progress, "General Infor- 
mation Yearbook - A Description of National Assessment Methodology, 
With Special Attention Given to Reading and Literature" (Washington, 
D.C. : U.S. Government Printing Office, 1972). Edward C. Bryant, 
et al.. Associations Between Educational Outcomes and Background 
Variables . (Rockville, Md. : Westat, Inc. , 1974) . 
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such a way that they presented the range of sizes found in 
the corporation. Finally, students were selected in each of 
the three grades (4, 8, and 12) by choosing every 16th name 
on lists of students, until the desired sample size was attained 
(excluding students absent or choosing not to participate) . 

Instrumentation for the reading portion of the study consisted 
primarily of items developed by NAEP. A paper-and-pencil , forced- 
choice format was used exclusively. Respondents were offered an 
"I do not know" alternative for each of the items and were given 
the option to select this alternative whenever they thought it to 
be most appropriate. 

Students also responded to items concerning their judgments 
about school in general, as well as their reading habits in parti- 
cular. Respondents provided data which indicated the educational 
level of their parents, sex, and racial heritage as well (see 
Attachment A) . 

In addition to information collected from students, data 
from state agency and 1970 Census records (Count IV) were used to 
construct additional independent variables concerning income 



differences among school districts and instructional differences 
among schools. 2 

How was the data analyzed and interpreted ? 

As stated earlier, the thrust of this study was to identify 
variables which related to reading performance. The main statis- 
tical tool employed in this investigation was multiple correlation! 
analysis. Also, factor analytic techniques were used to identify 
the structural properties of the data prior to establishing 
variables for analysis. 

The variance in reading achievement accounted for by multiple 
correlations was also partitioned using a technique called conunon- 
ality analysis. 

Commonality analysis allows one to divide the total variance 
explained into distinct portions which are (1) unique to one par- 
ticular variable or set of related variables, or (2) variations 
which only can be accounted for jointly across clusters of variables. 



2 

For more information concerning sample design and instru- 
mentation, see Indiana Educational Needs Assessment Project; A 
Report of Data Gathered in 1973 (Indiana Department of Public 
Instruction, 1974). For readers who are interested in the methods 
and summary findings of NAEP Reading Assessments, see the following 
publications compiled by the National Assessment of Educational 
Progress: Reading; Summary Data (Washington; U.S. Printing Office, 
1974), and Recipes, Wrappers, Reasoning and Rate; A Digest of the 
First Reading Assessment , (Washington, D. C. : U*S. Government 
Printing Office, 1974) . Reading; Released Exercises (Washington, 
D.C.: U.S. Government Printing Office, 1973), is a must publica- 
tion for those readers who are interested in NAEP results on an 
item-by-item basis* For an excellent overview of national assesss- 
ment goals and operations, see General Information Yearbook 
Washington, D.C. : U.S. Government Printing Office, 1974). 
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In this manner the effects of student background variables, 
variables based on judgment items, geographic and fiscal charac- 
teristics of school districts, and instructional characteristics 

of schools can be compared and contrasted as they related to each 

. . 3 

of the five dependent measures of reading achievement identified. 

Because of the exploratory nature of this study, no hypotheses 
are tested. To claim that the data supports hypotheses formed 
strictly on the basis of post hoc inspection is unwarranted. It 
seems appropriate, however, to observe whether or not this data 
is consistent with other major efforts to explain discrepancies 
in either school or student performance. In preparing the narrative 
sections of this report, an effort was made to compare these results 



3 

Cain and Watts in "Problems in Making Policy Inferences From 
the Coleman Report," American Sociological Review (1970), have 
faulted this technique for not treating variables which can be 
altered differently from those variables which cannot be affected 
by policy decisions. These statisticians claim that a "good" re- 
gression model is predicated upon sufficient theory so that 
magnitudes of predictable changes in the dependent variables can be 
ascertained. In his "Reply to Cain and Watts," American Sociologi - 
cal Review (1970) , Coleman asserted that educational variables have 
not reached a level of reliability where arguments against current 
procedures for not being abJLe to support ironclad predictions is 
appropriate. He also stated that a high degree of interdependency 
o£ variables included in his analysis would defeat attempts to use 
the size of regression coefficients to reflect direct relationship 
between corresponding predictors and the criterion. Also see 
Alexander Mood ' s introduction to A Study of the Attitude Toward 
Life of Our Nation's Students (Washington: U.S. Government Printing 
Office, 1973) . These writers concur that the commonality analysis 
model will not allow for ironclad predictions. But these procedures 
do indicate ways in v;hich variables interact in their influence on 
a criterion, and provide essential keys for further treatment through 
experimental studies. 
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4 

with conclusions and implications drawn from previous studies. 



What are the major findings ? 

Summary results are discussed in terms of two units of 

analysis, the student and the school. In both cases, predictors 

of variance have been determined by multiple linear regression 

5 

and partition of multiple correlation. In addition, some pre- 
liminary investigations using partial correlation analysis and 
analysis of variance within select subgroups of the population 
are highlighted. 



4 

See James Coleman, et al.. Equality of Educational Opportunity 
(National Center for Educational Statistics, 1966) , and George 
Mayeske, et al., A Study of Our Nation's Schools , and A Study of 
the Achievement of Our Nation's Students (Washington, D.C. : 
U.S. Office of "Education, HEW, 1970 and 1973, respectively). Many 
educators would benefit from a closer inspection of Christopher 
Jenks, Inequality — A Reassessment of the Effect of Family and 
Schooling in America . (New York: Basic Books, 1972) . Pivotal 
studies which utilized other than the "Coleman data base*' are 
listed in Attachment B. 

5 

These two units of observation have a tradition of serving 
different audiences. Analyses of differences in students have 
been of interest to persons involved with the instructional process 
in which analysis of differences between schools have been scru- 
tinized by policy makers at all levels of educational governance. 

The researchers used both units of analysis in treating this 
data in order to make a more reasoned comparison between these 
findings and more publicized reports of associations between school 
programs and student performance. The apparent lack of influence 
of instructional variables as reported separately by Coleman and 
Jenks was predicated upon the school, rather than the student, as the 
primary unit of observation. Other analyses — even of the Coleman 
data base — using the student as the primary unit of analysis tend 
to support the notion that a quality instructional program can 
influence a student's performance directly. 
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SUMMARY OF PRELIMINARY INVESTIGATIONS 



Before examining the main points, some preliminary findings 
are offered. Partial correlation analysis gave preliminary indi- 
cations of the independence of the effects of key variables. The 
only variables which showed significant interdependence in predict- 
ing reading ability were race, geographic region, and community 
type. This was not unexpected, since the racial mix of Indiana 
schools varies considerably over locale. 

A brief look at differences within select subpopulations also 
suggests that race, sex, and certain attitudes may interact in pre- 
dicting reading ability only within heritage groups according to the 
sex of the individual. When scores are broken out by subcategories 
of race, sex, parental education, and positive attitude toward formal 
educational process, the independent effect attributed to race is 
diminished. 

Items in the reading instrument were selected originally to 
reflect one of our specific objectives. Prior to these analyses, 
factor analytic techniques were employed to check whether or not 
items would cluster accordingly; only one strong factor appeared, 
however. More importantly, neither the one factor nor residual 
components could be defined by the criteria specified (subthemes) 
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in any of the four objectives. 



6 

A similar application of factor analysis has been performed 
on data collected by standardized reading tests. Each of the 
three tests included in the study claimed to differentiate between 
respondents on the basis of three to five of the expected outcomes 
of the reading process. A possibility of 11 separate factors 
existed prior to analysis; yet, only one factor emerged with an 
eigenvalue greater than 1. See John Oswald, William Lawlor, 
et al., "A Factor Analysis of Reading Skills at Grades 6, 9 and 
12, Using Variables From Three Standardized Reading Tests" (a 
paper presented at the T^eri.can Psychological Association Conven-- 
tion: New Orleans, 1974). 
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SUI4MARY OF RESULTS WITH STUDENT AS THE UNIT OF ANALYSIS 



The 77 variables used as predictors fall into four clusters: 
svudent background variables (race, parental education, and sex); 
geographic variables (region, type of community , family income 
within district) ; variables of student attitudes (judgments 
about reading, school, self, etc.); and instructional variables 
(varying patterns of school organization, staffing or existence 
of nontraditional curriculum) . 

In accounting for varying reading scores among students, the 

four clusters show a relatively high degree of independence from 

one another. This can be illustrated by the small ratio of common 

to unique portions of explained variance. It is noteworthy that 

the degree of independence increases as the students progress by 

grade level. 

In terms of the relative strength of the four clusters over 
grade levels, the associations appear to be remarkably stable. As 
expected, student judgment and student background clusters reach 
maximum association with reading at the 8th-grade level. 

Among the four clusters, only that cluster which deals with 
geographic location and average family income shows a negligible 
effect. These variables reach maximum association with reading 
at the senior level. The instructional and judgmental clusters 
maintain a strong and equally unique effect across the three grades. 
It is more difficult to separate the unique influence of student 
judgment variables from the unique influence of student background 
variables than is the case with instructional variables in combi- 
nation with other clusters. 



ERIC 



18 

11 



TABLE I. COMMONALITY ANALYSIS WITH STUDENT AS THE UNTT 
OF ANALYSIS, COMPONENTS EXPRESSED AS PERCENTAGES OF R^ . 



GRADE 4 



GRADE 8 



GRADE 12 



R' 
N 



.29749 
907 



Unique Components 

Ul (background variables) 
U2 (geographic variables) 
U3 (judgmental variables) 
U4 (instructional variables) 

Common Components 
C12 
C13 
C14 
C23 
C24 
C34 
C123 
C124 
C134 
C234 
C1234 



16.0 
2.1 
24 .2 
25.0 



3.3 
8.5 
6.4 
0.9 
1.7 
2.8 
1.9 
0.0 
5.1 
1.5 
0.7 



.25918 
1727 



23.0 
2.7 
30.0 
17.8 



4, 
10, 
3, 
0, 
2, 



0.8 
0.1 
2.4 
1.4 
0.8 
1.7 



.19148 
2161 



20.1 
8.1 
26.5 
24.2 



,6 
,8 
,2 
,2 
,5 
,8 



0.9 
5.8 
0.7 
0.7 
2.3 



TOTAL U 
TOTAL C 



67.3 
32.7 



73.5 
26.5 



78.9 
21.1 






GRADE 4 



GRADE 8 



GRADE 12 
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SOMMARY OF RESULTS WITH SCHOOL AC THE UNIT OF ANALYSIS 



Again^ commonality analysis was used to compare effects of 
the foiiv clusters de scr'ibod previously * However^ variables which 
previously described individual students (race and parental edu- 
cation and judgment variables ) were averaged over the school as 
was the dependent variable (reading scores). 

The similarity of results obtained with the school, rather 
than the student, as the focus of investigation was not expected. 
Most significantly, the cluster of instructional variables main- 
tain the strongest influence on reading levels and this effect 
increases over grade levels. On the other hand, the socioethnic 
characteristics of a school's student body — e.g., racial mix, mean 
parental education — appear to exert far less influence on reading 
scores than was thought would be the case. 

As was the case v;hen the student was the unit of analysis, 
the longer a student remains in school the more easily one can 
explain schoolwide reading skills on the basis of disjointed 
variables. The total per cent of explained variation which is 
unique to one of the four clusters of variables increases over 
grade levels. For example, it is difficult to separate the 
unique influence of students* attitudes from students' background 
characteristics at the 4th-grade level. By the senior year, these 
two clusters of variables do not share common variance in predict- 
ing differences in reading levels between schools. In addition, 
background characteristics of a school's student body, as well as 
more precisely identified school variables such as its location, 
size, average family income, etc., reach maximum association with 

20 

Er|c 13 



schoolwide reading performance at the 8th-grade level. By the 
senior year, that portion of unique variance attributed to race 
and/or geographic characteristics diminishes greatly. 
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TABLE II. COMMONALITY ANALYSIS WITH SCHOOL AS THE UNIT OF 
ANALYSIS; COMPONENTS EXPRESSED AS PERCENTAGES OF R . 



GRADE 4 GRADE 8 GRADE 12 



2 

R 


. 62097 


. 754 JO 


o n T 1 1 


N 


233 


157 


T T T 
111 


Unique Components 


6 . 8 


17 . 3 




Ul (background variables) 


7 • 0 


U2 (geographic variables) 


2.1 


12 . 8 


o o 

o • 3 


U3 (judgmental variables) 


21 . 4 


10 . 0 


23.0 


U4 (instructional variables) 


22 . 8 


27 . 9 


33.6 


Common Components 








C12 


- 1.3 


4 . 6 


1 . 3 


C13 


11.5 


0.8 


- 2.8 


C14 


2.8 


1.2 


- 0.5 


C23 


2.1 


2 . 6 


9 . 2 


C24 


0.6 


1.2 


-2.6 


C34 


11.1 


- 1.5 


7.6 
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What do these findings mean ? 

The most striking implication is that school programs do make 
a difference. Variables indicating the existence of select 
programs and instructional practices proved to be substantial pre- 
dictors of reading abilities for between-school comparisons as 
well as comparisons between individual students. This data also 
suggests that schools might reduce disproportionate levels of 
reading achievement between students of different ethnic backgro.und 
through differential alteration of instructional programming. 

Although school variables such as an ungraded program, differen- 
tiated staffing, etc., can be linked to higher levels of student 
reading ability, it does not mean that improvement in student per- 
formance can be readily attained. These variables might be 
indicators of a needed change in even more basic structures of school 
management. In any case, effective conversion to programs which 
encompass these concepts would require a rethinking on the part of 
teachers and administrators of both their philosophy of education 
and methods of instruction. 

The problem of successful alteration of instructional practices 
is further compounded by another major finding of this study. Other 
significant indicators of performance, such as students* background 
and attitudes, appear to be highly independent of program-oriented 
variables. Consequently, a sustaining and total effort to alter the 
significant correlates of reading must be multif aceted, recognizing 
the possibility that separate treatments may be needed when 
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modification of affective as well as cognitive behaviors are to 
be held as expected outcomes of the instructional process. 

These implications are not "cast in concrete." They should 
be regarded as hypotheses which call for further experimentation 
before one concludes that altering predictors will actually alter 
reading achievement. Firmer conclusions may be drawn when one 
assesses the impact this research has on the need to re-interpret 
results from similar investigations. Examples of how these 
findings are important in this vein are described in the following 
paragraphs . 

Most major studies investigating associations between schools 
and student outcomes (a) use the school rather than the student as 
the primary unit of observation, and (b) use a general measure of 
ability as criterion rather than a specific performance measure.. 
In regard to point (a) , this study employed both units of analysis 
An inspection of Chart I as compared to Chart II will show that 
the four clusters of variables maintain the same relative propor- 
tions even when averaged within schools. Thus, the apparent contr 
bution of instructional variable found in this investigation, but 
not in many others, cannot be explained simply as a result of 
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different units of analysis. In terms of point (b) , it has been 
noted that the National Assessment of Educational Progress claims 
that associations of student performance will vary significantly 
whenever specific performance indicators, instead of general 
ability, form the criteria for measurement. Since this study used 
NAEP measures, it is not surprising that these results tend to 
support their assumption. More specifically, this data showed a 
higher degree of independence among predictor variables than has 
been the case in similar investigations* For example, many re- 
searchers have concluded that it is impossible to distinguish 
between where background factors end and where school factors begin 
in their relative influence on student performance. Since similar 
data analysis procedures, as well as compatible criteria for selec- 
tion of predictors, were employed in this study, it would appear 
that the criterion selected has the most direct influence on the 
degree of expected intercorrelation among predictors. 

The major implication of these results, in contrast to some 
well-known studies, is that this data may be used to justify 
expenditure of resources for instructional improvement. On the 



7 

Although student variables may be averaged to provide school 
variables, there is no reason to expect variables collapsed in this 
manner to retain the same statistical relationships with each other. 
For example, if sex is a strong predictor in student analyses, then 
one might expect that it would disappear as a predictor of differ- 
ences between schools, since students are distributed evenly on 
the basis of sex in most public schools. Because many individual 
differences tend to "even out" V7hen explaining variation between 
schools, one might expect that the variables that do differentiate 
between schools account for a higher proportion of variance (R ) 
when the school, rather than the student, is the unit of analysis. 
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basis of similar investigations, many have concluded that quality 

"schooling" has little to do with the school itself. An unfortunate 

outgrowth of this line of thinking has been a maintenance of the 

status quo in the classroom so that educational resources could be 

directed toward implementing broader-range social policy. This 

study suggests that quality instruction might be the more effective 

key to unlock the mystery of improved educational outcomes. 

There are impressions that come through this data, hard and 

soft, in a variety of ways and which coincide with implications 

drawn in the past. Before concluding that the data runs counter 

to all investigations of recent vintage, some salient points of 

convergence are offered: 

-Broad indicators of attitude toward life account for a 
greater variation in reading ability than do indicators 
of attitude toward the specific task at hand. 

-Background factors appear to be differentially important 
for various minority groups, as well as dependent on the 
sex of the student (s) in question. 

-Deviations from expected scores for students with both 
parents educated beyond high school suggest that working 
mothers may have a less than positive impact on the child's 
reading performance quite aside from the impact of greater 
family income. 

-Racial heritage appears to account for slightly less than 
10 per cent of variation in scores after taking into account 
the influence of parental education and individual student 
characteristics . 

An implication for reading and measurement specialists might 
be drawn from preliminary analysis of instrumentation. Although 
items were chosen as representatives of various levels in a hier- 
archy of mental skills, factor analysis did not cluster items 
accordingly. This result suggests that more concrete characteristics 
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of the reading process, such as content or sentence structure, etc., 
may distinguish needed skills better than hierarchies of mental 
skills most often used to describe the expected outcomes of reading 
programs. 

Would a review of the entire report be beneficial ? 

Readers who are more interested in "how" this data was gener- 
ated than in what this data might mean from a practical point of 
view must review the entire text. For example. Chapter Two dis- 
cusses the construction and scaling of predictor and dependent 
variables, the proportion of variance explained by predictor vari- 
ables such as a function of sample size, and the possible effects 
of unexplained factors such as the students' innate abilities and 
differential dropout rates between ethnic groups. 

Individuals who are responsible for the design of large-scale 
instructional improvements may want to read the entire report to 
find out what instructional variables predicted best and how they 
interacted with other clusters of variables. Subsequent chapters 
will be more introspective and technical in nature, however. A 
firm grasp of the issues brought forth by these introductory remarks 
is needed if further analysis is to be of value. 

Those individuals responsible for large-scale needs assessment 
projects also could benefit from greater knowledge of this study. 
A related issue to which this study speaks is the question of 
whether or not inferential research is a productive step in the needs 
assessment process. While some persons within the education com- 
munity do see assessment as logically encompassing experimental 
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models of data analysis, others tend to view the assessment pro- 
cess more as vehicle toward public accountability than as a tool 
for behavioral research. The resulting debate has been a heated 
one .with practical and theoretical implications on both sides of 
the argument. 

A review of the entire report should illustrate the great amount 
of time which must be expended in selecting a sample, collecting 
data, defining variables, selecting appropriate statistical 
techniques, computerizing the data and statistics, and interpreting 
the results. Unless integrated data* bases are readily accessible 
to reduce the nuts and bolts constraints of data collection as well 
as statistical analysis, state assessment personnel should be con- 
tent with describing those things which appear to take place in our 
schools. 

In terms of theoretical limitations, all readers should be 
cautioned that the application of associational models to data 
collected by survey methods is a tenuous procedure. More experi- 
mental studies will always be needed to test the causal links 
suggested by studies such as this one. On the other hand, associa- 
tions uncovered by this investigation are not likely to be spurious 
since the sample was quite large and carefully stratified. The 
breadth of predictor variables considered allows for a variety of 
indepth investigations using the same body of data and subsequent 
hypothesis testing on comparable samples. 
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ATTACHMENT A 
Variable Definition 



Variables selected for this investigation fall into five 
groups: One group consists of five dependent measures of 
reading ability in addition to four different 'sets of independ- 
ent variables which reflect (1) background characteristics of 
students, (2) student attitudes toward school and reading, (3) 
geographic characteristics of the school district, and (4) charac- 
teristics of a school's instructional programs. 



READING 
ABILITY 




Income 
Location 
Size 



Instructional, 
organizational, 
and staffing 



FAIL 

DO NOT KNOW 



School 
District 



patterns as well 
as existence of 




curriculum innovations 




Ed. 

Sex 
Age 



Race 

Parental 



Judgments 
about 
school, 
reading and 
self 



School 
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Dependent Variables > Five separate dependent variables were 
defined from test items developed by National Assessment and 
administered in Indiana by the State Education Agency. Tv/o 
variables, FACT I and FACT 11, are each the number of items 
answered correctly from corresponding sets of items chosen on 
the basis of factor analysis. 

The frequency distribution of both FACT I and FACT II are 
skewed: FACT I high scores, and FACT II low scores. Since 
skewedi;iess can limit sensitivity, a third independent variable, 
TEST, was constructed. TEST is the number correct answers in a 
set of 10 items of moderate difficulty, and those which discri- 
minate best of the 78 items which comprise the original instrument. 

A fourth dependent variable, FAIL, is a dichotomous pass- 
fail measure with fail being defined as a score of less than 15. 
The fifth dependent variable, DON'T KN0V7, is the number of times 
a respondent selected the "I do not know" option in answering each 
of 78 items. 
Background Variables 

a) HER (heritage), dichotomous: (1) black; (0) other. 
Respondents who checked either the Spanish surname. 
Oriental or American Jndian categories were small 
in number and not included in summary analyses. 

b) ED (parental education), continuous: (1) neither 
parent attended high school; (2) one parent attended 
high school; (3) both attended high school; (4) one 
parent educated beyond high school; (5) both parents 
educated beyond high school 

c) Sex: (1) male; (0) female. 
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d) AD (Aga deviation from mean for grade level) (1) one 
standard deviation (SD) younger; (2) less than one 
SD; (3) one SD older. 

Attitudinal Variables 

a) AV021 (interest in reading) dichotomous: (1) dislike 
reading; (2) other. This variable was constructed from an 
item asking what the student thought about reading in 
school. One response is "I do not like to read." Other 
response alternatives indicate a more positive attitude ^ 
toward reading, hence they were lumped together in forming 
a dichotomous variable. 

b) AV022 (enjoyment of reading) continuous: (1) a waste of 
time; (2) boring; (3) enjoyable, but a chore; (4) enjoyable; 
(5) very enjoyable-. 

Tests of linearity reflected significant deviations from 
linearity and regressions on AV022 and its square provided 
the following transformations: 

DV3TEST T022= (2.16-0.30 AV022) AV022 

c) AV024 (frequency of library usage) continuous: (1) don't 
use; (2) 6 months ago; (3) 1 month ago; (4) this past 
week. 

Test of linearity supported the following transformation: 
DV3TEST TO24=(1.68-0.02 AV024) AV024 

d) AVO20 (like school) continuous: (1) I like it very much; 
(2) I like it; (3) neither like nor dislike; (4) many 
days I would like to .stay away; (5) if I could, I would 
quit school. 

e) AV056 ("people like me do not have any say about what 
the school does") dichotomous: (0) agree or not sure; 
(1) disagree or strongly disagree. 

f) AV055 (classroom rules) dichotomous: (1) rules judged 
to be too strict or not strict enough; (0) rules about 
right. 

g) AV056 (lack of classroom rules) dichotomous: (1) we have 
no rules; (2) other. 

h) AV086 (preferred kind of teacher) dichotomous: (1) one 
who sticks up for you; (0) other. 
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This variable as well as the next four variables 
were constructed by dichotomizing a categorical item. 



i) AV087, dichotomous: (1) 
.other . 

j) AV088, dichotomous: (1) 
other. 

k) AV089, dichotomous; (1) 
help; (0) other. 



one who understands you; (0) 
one who makes you work; (0) 
one who gives you lots of 



1) AVO90, dichotomous: (1) one who inspires you; (0) other. 

m) AV123 (availability of books in library) continuous: 

(1) never find materials I need; (2) seldom find materials 
I need; (3) usually find materials; (4) always find materials 
I need. 



n) AV124 (demands of society) dichotomous: (1) will not or 
probably be unable to live up to demands of society; (2) 
will live up to or probably will live up to demands of 
society. 

o) AV134 (perception of physical self) dichotomous: (1) do 
not like my body or wish I could change my body; (2) have 
no feelings one way or another, or are satisfied with body. 

p) AV136 (when beginning a new task) dichotomous: (1) I 
won't be successful; (2) I do well. 

q) AV177 (self in future) continuous: (1) almost sure I do 
not have a chance to succeed; (2) sometimes question if I 
will succeed; (3) am sure things will turn out well for me. 



Geographical Variables 

a) GVOINCOME (mean family income in school corporation) con- 
tinuous. This variable was thought to serve more as a 
physical descriptor of communities since family income 
was not available on a per student basis. 



Regional Variables 

Regional variables were constructed by creating a set of 
dummy variables from a categorical item with four options: 
North, Northwest, Indianapolis and South. 
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b) GVRl 10 0 0 

c) GVR2 0 10 0 

d) GVR3 0 0 1 0 
System Variables 

System variables were generated by constructing a set 
of dummy variables from a categorical item with five 
options describing community type: urban, suburban, large 
town, small city and rural. 



03 
U 

e) GVSl 1 

f) GVS2 0 

g) GVS3 0 

h) GVS4 0 

i) GVGSize (relative size for region and community type) 
dichotomous: (1) large; (2) small. 

School Variables 

Fourteen school variables were identified for analysis. They 
were derived through a principal component analysis of 52 separate 
descriptors of school organizational patterns, instructional 
patterns, staffing patterns, peripheral curriculum innovations or 
the existence of innovative program in mainline curriculum areas. 
Factor scores were computed for each factor to define variables for 
regression analysis ♦ 
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a) IVOl 

Computer Assist Instruction, 
Continuing Ed Program, 
Honors Program.... 

b) IV02 
Innovative Music, 
Innovative Art, 
Innovative Physical Ed.... 

c) IV03 

Parent Volunteers, 
Peer Tutors, 
Cadet Teachers.... 

d) IV04 

Innovative Home Economics 
Innovative Industrial Arts. . . . 

e) IV05 ** 
Ungraded Primary, 
Ungraded Intermediate. . . . 

f) IV06 

Lg -group, Sm-group Instruction 
Independent Study , 
Teacher Aides .... 

g) IV07 * . 
Business Ed . , 
Driver Ed ... . 



h) IV08 
Individual Program 

Instruction, 
Programmed Reading, 
Compensatory Ed. program. 

i) IV09 
Innovative Math, 
Innovative Social Studies 
Innovative Language Arts 
Innovative Science .... 

j) IVOlO * 

Voc-Tech Ed program. . . . 

k) AVOia 

Ungraded high school. 
Ungraded middle school... 

1) IV012 

Reading lab. 
Differentiated staff. 
Flexible schedule.... 

m) AV013 

Innovative Foreign 

Language Arts, 
Foreign Language .... 

n) AV014 

Drug Program. . . . 



High school only 
* Elementary school only 
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